King County Housing Prices

This project aims to explore data on housing prices in an effort to design a model for estimating pricing and other behaviors of the housing market, specifically within King County.

suppressPackageStartupMessages
(
  {
    #install or load required packages into the workspace
    package_list <- c('tidyverse', 'knitr', 'plotly','RColorBrewer','sqldf')
    non_installed <- package_list[!(package_list %in% installed.packages()[,"Package"])]
    if(length(non_installed)) install.packages(non_installed)
    #library('tidyverse')
    library('plotly')
    library('RColorBrewer')
    library('sqldf')
    library('knitr')
  }
)
knitr::opts_chunk$set(message = F, warning = F, strip.white = F, tidy = T)

load in the required packages

#load dataset from project folder
suppressWarnings({housing_data <- read.csv("kc_house_data.csv", stringsAsFactors = F)})
#remove unused columns and normalize the data
housing_data_rev <- housing_data[, 3:21]

load in dataset and remove unused features

#collect a random sampling of the dataset ordered by price
train_vals <- sample(dim(housing_data)[1], 1000)
test_vals <- sample(dim(housing_data)[1], 200)
#full test set?
#test_vals <- !(1:dim(housing_data)[1] %in% train_vals)
housing_train_data <- as.data.frame(housing_data_rev[train_vals,])
housing_train_data <- housing_train_data[order(-housing_train_data$price),]
housing_test_data <- as.data.frame(housing_data_rev[test_vals,])
housing_test_data <- housing_test_data[order(-housing_test_data$price),]
sqldf("Select price As Pricing, grade As Rating, condition As Condition, sqft_lot As Sq_Footage 
       From housing_train_data
       Limit 20")

This dataset is very large, so in order to get a better visual representation of the distribution of the data, a sample of 1000 randomly chosen rows is used instead of the whole dataset (ordered by price, as this is the dependent variable). The sample indexes are selected at runtime every time the script runs, so the data and analysis are different each time the script is run. Because the sampling is random and the dataset is homogenous and static, the 1000 rows chosen should always be representative of the nature of the whole dataset.

#Look at the distributions:
p1 <- plot_ly() %>% add_markers(name = "Pricing", x = 1:1000, y = housing_train_data$price) %>% layout(title = "King County Housing Feature Distributions")
p2 <- plot_ly() %>% add_markers(name = "Rating v Pricing", x = housing_train_data$grade, y = housing_train_data$price)
p3 <- plot_ly() %>% add_markers(name = "Condition v Pricing", x = housing_train_data$condition, y = housing_train_data$price)
p4 <- plot_ly() %>% add_markers(name = "Sq. Footage v Pricing", x = housing_train_data$sqft_lot, y = housing_train_data$price)
subplot(p1, p2, p3, p4, nrows = 2)

Looking at the distributions of prices by magnitude, it appears to follow a power law distribution that isn’t mirrored in the distributions of the other features seen here. Assuming that the features in this dataset capture most of the factors that influence the pricing of a home, some linear combination of the features could give a reasonable estimate for the pricing of an individual house. Since the distribution is closer to a power law, two seperate linear functions should give a pretty good estimate by accounting for the ‘knee’ in the pricing distribution. First though, we’ll try fitting a single line to the data and see what kind of error comes out.

#split data for training
#test general linear model on relevant quantitative data
housing_train_price <- housing_train_data[, "price"]
housing_test_price <- housing_test_data[, "price"]
glm_mod <- glm(price ~ bedrooms + bathrooms + sqft_living + sqft_lot + condition + grade + yr_built + yr_renovated, data = housing_train_data)
price_mod <- predict(glm_mod, newdata = housing_test_data)
price_est_plotly <- plot_ly() %>% 
  add_markers(name = "Test Set Price", x = 1:length(housing_test_price), y = housing_test_price) %>%
  add_markers(name = "Linear Model est", x = 1:length(housing_test_price), y = price_mod) %>%
  add_lines(name = "Magnitude Difference", x = 1:length(housing_test_price), y = price_mod - housing_test_price) %>%
  layout(title = "Attempting Linear Regression Model </br> 
         for Pricing Homes in King County </br>")
price_est_plotly

Given that the distribution of prices was already known to follow a power law distribution, the result of this linear estimate isn’t very surprising. The linear fit is good for the long tail, but the whole is corrupted by trying to fit the collection of values above the elbow.